You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Royce Williams <ro...@gmail.com> on 2010/04/02 23:45:57 UTC
skipping dynamic tests for ISP's own dynamic networks?
What is the optimal configuration (local.cf or other) for an ISP's
MSAs to prevent unauthenticated dynamic-IP customers from triggering
dynamic tests, but still benefiting from general filtering?
I was hoping for a magical 'mua_networks' option, which let me
enumerate the IP space that my users submit from, and automatically
exempt them from DOS_OE_TO_MX, etc., but I haven't been able to find
anything like that.
>From my reading of the .conf manpage, the TrustPath page, and the
archives (see references below), I've tentatively concluded that I
will need to have some local rule overrides on all of my MSAs for any
rule or meta-rule that detects dynamic-looking hostnames ... but that
seems high-maintenance locally as well as a lot of duplicated work for
other SA users.
Note also that my MTAs and MSAs are separate farms, so having separate
local.cfs for each is 100% feasible.
To point out the obvious: because my customers may be trojaned (or
decided to become spammers), I cannot assume that they aren't forging
headers - but I know that they are *supposed* to have hostnames that
look dynamic. Also, for legacy reasons, lots of them are
authenticated only by what IP space they're coming from (not SMTP
AUTH, etc.)
I'm a long-time user, first-time poster; any help would be much
appreciated. I suspect that this is a well-solved issue, and I just
failed to come up with the right Google search for it.
Royce Williams
References:
http://spamassassin.apache.org/full/3.3.x/doc/Mail_SpamAssassin_Conf.html
http://wiki.apache.org/spamassassin/TrustPath
http://old.nabble.com/ALL_TRUSTED-and-DOS_OE_TO_MX-td15659736.html
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Royce Williams <ro...@gmail.com>.
On Fri, Apr 2, 2010 at 1:45 PM, Royce Williams <ro...@gmail.com> wrote:
> What is the optimal configuration (local.cf or other) for an ISP's
> MSAs to prevent unauthenticated dynamic-IP customers from triggering
> dynamic tests, but still benefiting from general filtering?
Sorry, 'unauthenticated' is misused here; I meant not authenticated
via SMTP auth, only by IP space.
Royce
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Royce Williams <ro...@gmail.com>.
On Tue, Apr 6, 2010 at 1:01 PM, Royce Williams <ro...@gmail.com> wrote:
> Also, I think that an example snippet of.cf illustrating and briefly
> explaining each of the three _networks options might be in order, and
> might make the reading, re-reading, and re-reading of the docs a
> little less painful. Writing one will also demonstrate that I've
> correctly grokked what's been going on here. :-) I'll take a stab at
> one.
OK, here is my first pass at a *_networks example. Feedback welcome.
I've included comments and whitespace, to make it easy to digest
for others trying to wrap their head around this in the future.
If you have a minor correction, please send it to me off-list and I
will compile. If you spot a fundamental issue, please point it out
on-list (to quickly get consensus).
I'm specifically interested in getting other server type examples
so that lots of bases are covered for educational purposes.
I also need some help understanding these two adjacent paragraphs of
the msa_networks explanation so that I can fold them in:
"All relays found in the message headers after the MSA relay will take
on the same trusted and internal classifcations as the MSA relay itself,
as defined by your trusted_networks and internal_networks
configuration."
"For example, if the MSA relay is trusted and internal so will all of
the relays that precede it."
The use of the word "after" in the first paragraph and "precede" in the
second paragraph confuse me. "after" appears to be meant in the
chronological sense, and "precede" appears to be the order as presented
when viewing the headers -- is this correct? Logically, this would
make more sense than the reverse.
OK, here's my draft example.
#-----------------------------------------------------------------------
# This example illustrates the interaction between network
# configuration parameters (*_networks) in SpamAssassin's .cf files.
#
# Most SA setups will not include all of these types of servers, but
# they are all included for demonstration purposes.
#
# Given the following networks and systems:
#
# 10.0.0.0/24 - Dedicated MTA/MXes, handling inbound mail.
# 10.1.1.0/24 - Dedicated MSAs at the border, handling outbound mail.
# 10.2.0.0/16 - Customer dynamic/dialup/statics allowed to send mail.
# 10.3.3.0/24 - A trusted third-party relay (like Postini).
# 10.4.4.0/24 - A set of hybrid MX/MSA systems.
# 10.5.5.5 - A legacy inbound mail server used for ETRN.
# 10.6.6.6 - intermediate outbound server for internal workstations.
#
# ... then the local.cf could contain:
#----------------------------
# trusted_networks
#----------------------------
#
# Contains: Relays that may relay spam, but will never originate it,
# and will never forge headers. This must also contain *all*
# members of internal_networks.
#
# Should not contain: Any system that could originate spam
# or forge headers.
#
# Effects: DNSBL checks will never query for hosts on these networks.
#
# Dedicated MXes.
trusted_networks 10.0.0.0/24
# Dedicated MSAs.
trusted_networks 10.1.1.0/24
# Postini.
trusted_networks 10.3.3.0/24
# Hybrid MX/MSA systems.
trusted_networks 10.4.4.0/24
# ETRN server (baton).
trusted_networks 10.5.5.5
# handoff.example.net - our internal intermediate relay.
trusted_networks 10.6.6.6
#----------------------------
# msa_networks
#----------------------------
#
# Contains: Your relay that accept mail from your authenticated users,
# by IP list, SMTP AUTH, POP-before-SMTP, etc.
#
# Should not contain: Multipurpose MSA/MX systems or any other
# systems that accept non-authenticated/verified submission.
#
# Effects: Headers added by subsequent relays inherit the "trust"
# and "internal" status of the MSA. Also, messages submitted by
# authenticated/verified users to these systems will not be
# subject to tests designed to detect dynamic IP DNS names, direct
# messages from clients to MXes, etc. [NEED MORE HERE]
#
# Dedicated MSAs.
msa_networks 10.1.1.0/24
# Our internal intermediate relay (handoff.example.net)
10.5.5.5
#----------------------------
# internal_networks
#----------------------------
#
# Contains: MXes or internal relays.
#
# Should not contain: trusted hosts that are not MXes or internal
# relays. For example, MSAs should not be included here.
#
# Effects: Used when checking dialup or dynamic IP address blocklists,
# in order to detect direct-to-MX spamming. [NEED MORE HERE]
#
# Dedicated MXes.
internal_networks 10.0.0.0/24
# ETRN server (baton).
internal_networks 10.5.5.5
#
# Note that none of these parameters contains the end-user network space
# (10.2.0.0/16).
#
# (end of example)
#-----------------------------------------------------------------------
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Charles Gregory <cg...@hwcn.org>.
On Thu, 15 Apr 2010, Royce Williams wrote:
> I will also file a bug to suggest updates to the *_networks language
> that is in direct contradiction to the advice in other parts of this
> thread.
One thing I might add: It seemed to me that at certain points in the
discussion there was confusion as to whether the status of the mail server
running spamassassin was influenced by the mua_networks setting. I believe
some language in the docs would be appropriate to distinguish any
'*_networks' settings that would be appropriate for the server *running*
SA.
Ie. The language about 'mua_networks' makes sense when you realize it is
about 'trusting' *another* server on your network that has handled the
dynamic IP's, as opposed to having SA simlpy 'trust' (as in
'trusted_networks' the dynamic IP's from which it directly receives
outgiong mail.
- C
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Royce Williams <ro...@gmail.com>.
On Mon, Apr 12, 2010 at 12:20 PM, Kris Deugau <kd...@vianet.ca> wrote:
> Royce Williams wrote:
>>
>> From the documentation, msa_networks designates those servers that
>> accept only authenticated messages, regardless of type. I'm the new
>> guy on the list, and have some catching up to do with learning how the
>> *_networks directives work, but the evidence is mounting that if MSAs
>> listed in msa_networks can't tell that they're in msa_networks, then
>> msa_networks does not work as documented.
>
> Can you provide a more detailed explanation of the system(s) that are
> misbehaving? I've got three different sets of servers with three different
> SA integration methods for outbound mail, and all three Do The Right
> Thing(TM) with the same *_networks settings.
>
> It sounds like you've got some factor interfering, but aside from DTDW
> ("Damn Thing Doesn't Work") errors telling us that *something* is broken,
> it's hard to see where the trust path is breaking down.
>
> I think from your 10/8 example you've got your head pretty much wrapped
> around the semantics of the configuration options (the only thing I would
> set differently would be to add services like Postini to internal_networks,
> because you want eg Spamhaus rules to trigger on the IP that relayed to
> Postini, not the Postini filter server IP).
Kris, Thanks for the feedback, generally and about my example. I will
try to do more sleuthing along the trust path and report back when I
have something definitive to say or get stuck.
I will also file a bug to suggest updates to the *_networks language
that is in direct contradiction to the advice in other parts of this
thread.
Royce
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Henrik K <he...@hege.li>.
On Sat, Apr 10, 2010 at 09:02:52AM -0800, Royce Williams wrote:
> On Sat, Apr 10, 2010 at 7:25 AM, Henrik K <he...@hege.li> wrote:
> > On Sat, Apr 10, 2010 at 07:02:55AM -0800, Royce Williams wrote:
> >> On Sat, Apr 10, 2010 at 6:49 AM, Royce Williams
> >> <ro...@gmail.com> wrote:
> >> > * Create a mua_networks option. This would only need to interact with
> >> > msa_networks, and would allow msa_networks systems to become
> >> > self-aware. If a server is in msa_networks, and it sees someone
> >> > connecting from a mua_network, then it would treat them as
> >> > authenticated.
> >>
> >> Hmm. This assumes that SA is aware of its own IP addresses, and
> >> mua_networks would only be needed to determine where the border is.
> >> Is this true? If not, and SA is only indirectly aware of its own IPs
> >> by virtue of looking at headers, that might be a factor.
> >
> > Why are you interested in mua_networks, since it's already stated that just
> > adding the ranges in internal_networks suffices? The only difference would
> > be that it won't be recursive, but I doubt dial-ups would be relays for
> > other dial-ups.
>
> Thanks for your patience, Henrik. I am trying to be clear, but I'm
> still learning the vocabulary here.
>
> As I see it, there are three problems with listing those ranges.
>
> First, from my testing, this method does not keep dial-ups and dynamic
> IPs from triggering direct-to-MX rules, while at the same time
> continuing to filter them for content and being suspicious of their
> possibly-forged headers.
Then you have misconfigured and/or are still using 3.2.5 which had broken
helo/dynamic rules (the rules looked at untrusted border instead of
external).
> For example, with my own dynamic IP at home as a member of *only*
> internal_networks, and with everything else configured as specified in
> this thread and the docs, HELO and RDNS enforcement rules are triggered.
>
> Second, if it did work, it would be in direct contradiction with one
> or both of these firmly-stated documentation items:
>
> "Trusted relays that accept mail directly from dial-up connections
> should not be listed in internal_networks. List them only in
> trusted_networks."
It's not surprising that even SA developers can get these wrong.
Grep the rules. If they look at X-Spam-Relays-External, then they are
correct.
spamassassin -t -D < msg 2>&1 | grep X-Spam-Relays
If your dial-up shows up in X-Spam-Relays-External, then you have
misconfigured.
These are the facts.
> and
>
> "Every entry in internal_networks must appear in trusted_networks; in
> other words, internal_networks is always a subset of the trusted set."
>
> Third, if this is the recommended method, it means that msa_networks
> does not function as documented.
>
> ISP end customer systems are a hybrid -- legitimately using Outlook
> and coming from dynamics, but also sometimes a spam source. I am
> trying to express that "hybridness" with SpamAssassin. I have tested
> all of the methods documented and proposed on the list, and they do
> not express this.
>
> I'm not dead-set on the idea of mua_networks. If there is a better,
> scalable, benefits-everybody way to solve what I believe to be a basic
> problem, I am interested. I also want my effort here to benefit
> others -- which means that I want to help update the documentation to
> reflect reality, or to help to adjust reality to match the
> documentation.
I think this will be my last contribution, since this goes on and on. :-)
Use spamassassin debug, look at X-Spam-Relays-*. Look at rules. Maybe even
look at code. It's all there. Sadly documentation can be outdated or
misleading.
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Royce Williams <ro...@gmail.com>.
On Sat, Apr 10, 2010 at 7:25 AM, Henrik K <he...@hege.li> wrote:
> On Sat, Apr 10, 2010 at 07:02:55AM -0800, Royce Williams wrote:
>> On Sat, Apr 10, 2010 at 6:49 AM, Royce Williams
>> <ro...@gmail.com> wrote:
>> > * Create a mua_networks option. This would only need to interact with
>> > msa_networks, and would allow msa_networks systems to become
>> > self-aware. If a server is in msa_networks, and it sees someone
>> > connecting from a mua_network, then it would treat them as
>> > authenticated.
>>
>> Hmm. This assumes that SA is aware of its own IP addresses, and
>> mua_networks would only be needed to determine where the border is.
>> Is this true? If not, and SA is only indirectly aware of its own IPs
>> by virtue of looking at headers, that might be a factor.
>
> Why are you interested in mua_networks, since it's already stated that just
> adding the ranges in internal_networks suffices? The only difference would
> be that it won't be recursive, but I doubt dial-ups would be relays for
> other dial-ups.
Thanks for your patience, Henrik. I am trying to be clear, but I'm
still learning the vocabulary here.
As I see it, there are three problems with listing those ranges.
First, from my testing, this method does not keep dial-ups and dynamic
IPs from triggering direct-to-MX rules, while at the same time
continuing to filter them for content and being suspicious of their
possibly-forged headers. For example, with my own dynamic IP at home
as a member of *only* internal_networks, and with everything else
configured as specified in this thread and the docs, HELO and RDNS
enforcement rules are triggered.
Second, if it did work, it would be in direct contradiction with one
or both of these firmly-stated documentation items:
"Trusted relays that accept mail directly from dial-up connections
should not be listed in internal_networks. List them only in
trusted_networks."
and
"Every entry in internal_networks must appear in trusted_networks; in
other words, internal_networks is always a subset of the trusted set."
Third, if this is the recommended method, it means that msa_networks
does not function as documented.
ISP end customer systems are a hybrid -- legitimately using Outlook
and coming from dynamics, but also sometimes a spam source. I am
trying to express that "hybridness" with SpamAssassin. I have tested
all of the methods documented and proposed on the list, and they do
not express this.
I'm not dead-set on the idea of mua_networks. If there is a better,
scalable, benefits-everybody way to solve what I believe to be a basic
problem, I am interested. I also want my effort here to benefit
others -- which means that I want to help update the documentation to
reflect reality, or to help to adjust reality to match the
documentation.
Royce
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Benny Pedersen <me...@junc.org>.
On lør 10 apr 2010 17:25:56 CEST, Henrik K wrote
> but I doubt dial-ups would be relays for other dial-ups.
you should really know better here
--
xpoint http://www.unicom.com/pw/reply-to-harmful.html
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Henrik K <he...@hege.li>.
On Sat, Apr 10, 2010 at 07:02:55AM -0800, Royce Williams wrote:
> On Sat, Apr 10, 2010 at 6:49 AM, Royce Williams
> <ro...@gmail.com> wrote:
> > * Create a mua_networks option. This would only need to interact with
> > msa_networks, and would allow msa_networks systems to become
> > self-aware. If a server is in msa_networks, and it sees someone
> > connecting from a mua_network, then it would treat them as
> > authenticated.
>
> Hmm. This assumes that SA is aware of its own IP addresses, and
> mua_networks would only be needed to determine where the border is.
> Is this true? If not, and SA is only indirectly aware of its own IPs
> by virtue of looking at headers, that might be a factor.
Why are you interested in mua_networks, since it's already stated that just
adding the ranges in internal_networks suffices? The only difference would
be that it won't be recursive, but I doubt dial-ups would be relays for
other dial-ups.
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Royce Williams <ro...@gmail.com>.
On Sat, Apr 10, 2010 at 6:49 AM, Royce Williams
<ro...@gmail.com> wrote:
> * Create a mua_networks option. This would only need to interact with
> msa_networks, and would allow msa_networks systems to become
> self-aware. If a server is in msa_networks, and it sees someone
> connecting from a mua_network, then it would treat them as
> authenticated.
Hmm. This assumes that SA is aware of its own IP addresses, and
mua_networks would only be needed to determine where the border is.
Is this true? If not, and SA is only indirectly aware of its own IPs
by virtue of looking at headers, that might be a factor.
Royce
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Kris Deugau <kd...@vianet.ca>.
Royce Williams wrote:
> From the documentation, msa_networks designates those servers that
> accept only authenticated messages, regardless of type. I'm the new
> guy on the list, and have some catching up to do with learning how the
> *_networks directives work, but the evidence is mounting that if MSAs
> listed in msa_networks can't tell that they're in msa_networks, then
> msa_networks does not work as documented.
Can you provide a more detailed explanation of the system(s) that are
misbehaving? I've got three different sets of servers with three
different SA integration methods for outbound mail, and all three Do The
Right Thing(TM) with the same *_networks settings.
It sounds like you've got some factor interfering, but aside from DTDW
("Damn Thing Doesn't Work") errors telling us that *something* is
broken, it's hard to see where the trust path is breaking down.
I think from your 10/8 example you've got your head pretty much wrapped
around the semantics of the configuration options (the only thing I
would set differently would be to add services like Postini to
internal_networks, because you want eg Spamhaus rules to trigger on the
IP that relayed to Postini, not the Postini filter server IP).
-kgd
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Royce Williams <ro...@gmail.com>.
On Sat, Apr 10, 2010 at 12:41 AM, Henrik K <he...@hege.li> wrote:
> On Fri, Apr 09, 2010 at 05:43:24PM -0400, Kris Deugau wrote:
>>
>>>> I would think that in this case the dynamic address blocks would need to
>>>> be explicitly defined.
>>>
>>> That's why I starting this thread by saying that I went hunting for a
>>> "mua_networks" equivalent, and couldn't find one.
>>
>> OK, think about this: What do you do about relay IPs outside your
>> network, from which your customers are sending mail through your MSA via
>> SMTP AUTH? There's a good chance they're listed on eg Spamhaus PBL -
>> and there's *no* way you'll ever predict them.
Agreed. I'm trying to solve the problem for my legacy-support MSAs,
which are only supporting non-SMTP-AUTH clients.
>From the documentation, msa_networks designates those servers that
accept only authenticated messages, regardless of type. I'm the new
guy on the list, and have some catching up to do with learning how the
*_networks directives work, but the evidence is mounting that if MSAs
listed in msa_networks can't tell that they're in msa_networks, then
msa_networks does not work as documented.
I know that folks are mostly straight SMTP AUTH nowadays. I still
have to support IP lists, and the documentation says that they are
supported. If this isn't true, then I'd like to help to make it true,
or request that the documentation be updated.
> If one really can't make it work by the proper msa_networks way (making sure
> all the auth Received: magic is there), then probably the only option is to
> use a POPAuth[1] style plugin (somewhat deprecated since it doesn't even
> recognize "internal"). I'm sure there's plugin ways to add to the internal
> list if the sender is authenticated, but still it probably would be more
> beneficial to just change the MSA configuration to supported.
>
> [1] http://wiki.apache.org/spamassassin/POPAuthPlugin
If I have to do something to duplicate the intended functionality of
msa_networks, it would be more efficient (and benefit the user
community at large) if the problem with msa_networks is addressed
globally. In other words, I do not believe that I am misconfiguring
my SpamAssassin.
msa_networks systems are not currently aware of themselves. It sounds
like SMTP AUTH connections are already covered, but other
authenticated methods (specifically, "IP list" methods like sendmail's
access file are not currently working with msa_networks. So, some
possible options are (thinking out loud):
* Insert a pseudo-header with MIMEDefang, etc. This is sub-optimal, as
it becomes cross-site whack-a-mole for different platforms.
* POPAuth. This is inefficient, given that we're going to lengths to
recreate knowledge that we already know - that we allow everyone from
certain networks to relay.
* Create a mua_networks option. This would only need to interact with
msa_networks, and would allow msa_networks systems to become
self-aware. If a server is in msa_networks, and it sees someone
connecting from a mua_network, then it would treat them as
authenticated.
* Other?
I am not approaching this problem with a sense entitlement. If the
core developers are interested in the mua_networks method, or know of
another way to tackle the problem, I would like to help. I know some
Perl, and could probably figure something out, but do not want to go
to the effort if it will not be accepted to benefit the whole
community.
Royce
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Henrik K <he...@hege.li>.
On Fri, Apr 09, 2010 at 05:43:24PM -0400, Kris Deugau wrote:
>
>>> I would think that in this case the dynamic address blocks would need to
>>> be explicitly defined.
>>
>> That's why I starting this thread by saying that I went hunting for a
>> "mua_networks" equivalent, and couldn't find one.
>
> OK, think about this: What do you do about relay IPs outside your
> network, from which your customers are sending mail through your MSA via
> SMTP AUTH? There's a good chance they're listed on eg Spamhaus PBL -
> and there's *no* way you'll ever predict them.
If one really can't make it work by the proper msa_networks way (making sure
all the auth Received: magic is there), then probably the only option is to
use a POPAuth[1] style plugin (somewhat deprecated since it doesn't even
recognize "internal"). I'm sure there's plugin ways to add to the internal
list if the sender is authenticated, but still it probably would be more
beneficial to just change the MSA configuration to supported.
[1] http://wiki.apache.org/spamassassin/POPAuthPlugin
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Kris Deugau <kd...@vianet.ca>.
Royce Williams wrote:
> On Fri, Apr 9, 2010 at 3:46 AM, RW <rw...@googlemail.com> wrote:
>> msa_networks defines the MSA by IP address. If SA runs on an MSA its
>> address is unlikely to be in the received headers. In that case SA has
>> no way of distinguishing an MSA from an MX server.
>
> Yes! That's what Daryl was referring to here
>
> http://old.nabble.com/ALL_TRUSTED-and-DOS_OE_TO_MX-td15659736.html
>
> ... where he says:
>
> "So if (and I'll admit I don't think this occurred to me before) you're
> running SA on outgoing mail on your MSA right after you receive it (it's
> not relayed to an intermediate machine) SA can't detect the MSA and the
> whole msa_networks thing doesn't work."
*nod* I'm seeing the logic there... now I'm trying to figure out what
kind of magic chicken I must have sacrificed to get things to work here.
<g>
>> I would think that in this case the dynamic address blocks would need to
>> be explicitly defined.
>
> That's why I starting this thread by saying that I went hunting for a
> "mua_networks" equivalent, and couldn't find one.
OK, think about this: What do you do about relay IPs outside your
network, from which your customers are sending mail through your MSA via
SMTP AUTH? There's a good chance they're listed on eg Spamhaus PBL -
and there's *no* way you'll ever predict them.
> Henrik and RW have both suggested that I should put my customer-only
> MSAs into msa_networks and internal_networks (which implies
> trusted_networks). I can state definitively that in this setup, all
> of the you-look-like-a-MUA rules (RDNS, Outlook, etc.) are happily
> applied to my dialup customers, which is consistent with RW's
> statement above.
Ahhh, here's a code comment:
Mail/SpamAssassin/Conf/Parser.pm: line ~1040
# validate trusted_networks and internal_networks, bug 4760.
# check that all internal_networks are listed in trusted_networks
# too. do the same for msa_networks, but check msa_networks against
# internal_networks if trusted_networks aren't defined
So msa_networks *may* be a subset of internal, but it's not required,
and not quietly forced either *unless* trusted_ isn't defined.
All I can say is "WorksForMe(TM)". :/ I have three different systems
with several different SA versions, and several integration methods...
and none of them trigger on direct-to-MX-ish rules inappropriately.
One is our primary mail cluster; the SA filter subcluster is used for
both inbound scanning and outbound. On the outbound side, a Postfix
subcluster calls SA via custom Postfix content filter (which does not
generate a Received: header). Currently it's running SA3.3.1, but the
trust config was set with 3.2.5 - the last update just added a machine
to msa_networks when I discovered the occasional customer mail tripping
the outbound filter with ... yep, direct-to-MX-ish rules. (The Speed
Dial SMTP proxy running on that machine added a Received: header while
streaming the message to the "real" MSA cluster, and the message ended
up looking like direct-to-MX because there was a trusted non-MSA host in
between the MUA and SA.)
SA scan result:
spamd: result: . 0 - AWL,BAYES_20,NO_RECEIVED,NO_RELAYS
The other two are legacy all-in-one domain-hosting servers. One runs SA
via amavisd-new as a Postfix content filter (post-queue; no SMTP-time
rejections). The other calls SA from MIMEDefang. .... hmm, now that I
poke and think, both systems likely pass either a real live Received:
header (Postifx+Amavis) or a synthetic one (MIMEDefang) to SA.
SA scan results:
Amavis:
X-Spam-Status: No, score=-101.378 tagged_above=-9999 required=6.31
tests=[ALL_TRUSTED=-1.8, AWL=-1.298, BAYES_50=0.001,
TVD_SPACE_RATIO=1.719, USER_IN_WHITELIST=-100]
MIMEDefang:
X-Spam-Score: -99.21 () req=5 BAYES_50,T_RP_MATCHES_RCVD,USER_IN_WHITELIST
(Both delivered to test accounts I set up local to each system.)
Here's an obfuscated version of the live config (used on all three systems):
clear_trusted_networks
# core servers
trusted_networks 192.168.0.0/24
# legacy the first
trusted_networks 192.168.1.0/26
trusted_networks 192.168.3.0/26
# legacy the second
trusted_networks 192.168.4.232/29
# and third
trusted_networks 192.168.5.0/26
# inherited Plesk
trusted_networks 192.168.6.160/27
# colo(ish) customer server
trusted_networks 192.168.7.122/32
# postini. *sigh*
trusted_networks 64.18.0.0/20
# messagelabs (amsterdam)
trusted_networks 195.245.231.0/24
# customer's third-party webhost I'm willing to trust
# server.superhost2.nl
trusted_networks 91.192.36.238/32
# willing to believe UBC has someone halfway
# competent running their mail systems...
trusted_networks 137.82.45.0/28
# observed: .1, .5, .7, .15
# rDNS shows .1 -> .17 or so as MTA-ish
## internal
clear_internal_networks
internal_networks 192.168.0.21
internal_networks 192.168.0.22
internal_networks 192.168.0.23
internal_networks 192.168.0.24
internal_networks 192.168.0.119
# put Postini here so eg Spamhaus rules hit properly - otherwise
# they'll run on the Postini relay IP.
internal_networks 64.18.0.0/20
# gotta get RBL hits on these guys' headers...
internal_networks 91.192.36.238/32
# this should ONLY be machines that act as
# outbound relays; never inbound.
clear_msa_networks
msa_networks 192.168.0.17
msa_networks 192.168.0.18
msa_networks 192.168.0.19
# this machine handles "speed dial" mail; it adds
# a Received: header but not one of much use.
msa_networks 192.168.0.9
-kgd
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Royce Williams <ro...@gmail.com>.
On Fri, Apr 9, 2010 at 3:46 AM, RW <rw...@googlemail.com> wrote:
> On Fri, 9 Apr 2010 10:09:35 +0300
> Henrik K <he...@hege.li> wrote:
>
>> On Thu, Apr 08, 2010 at 10:26:27PM -0800, Royce Williams wrote:
>> > >
>
>> > Maybe I'm having a vocabulary problem. My MSAs are really also
>> > MTAs - they receive mail from the customer, do an MX lookup on the
>> > destination domain, and relay. But they are not MXes in that they
>> > do not receive mail from foreign MTAs.
>>
>> Read and re-read "msa_networks" documentation. IMHO it's very clearly
>> defined. It's just an extender for *_networks.
>>
> I think he may have put his finger on the problem in a previous post.
>
> msa_networks defines the MSA by IP address. If SA runs on an MSA its
> address is unlikely to be in the received headers. In that case SA has
> no way of distinguishing an MSA from an MX server.
Yes! That's what Daryl was referring to here
http://old.nabble.com/ALL_TRUSTED-and-DOS_OE_TO_MX-td15659736.html
... where he says:
"So if (and I'll admit I don't think this occurred to me before) you're
running SA on outgoing mail on your MSA right after you receive it (it's
not relayed to an intermediate machine) SA can't detect the MSA and the
whole msa_networks thing doesn't work."
> I would think that in this case the dynamic address blocks would need to
> be explicitly defined.
That's why I starting this thread by saying that I went hunting for a
"mua_networks" equivalent, and couldn't find one.
Henrik and RW have both suggested that I should put my customer-only
MSAs into msa_networks and internal_networks (which implies
trusted_networks). I can state definitively that in this setup, all
of the you-look-like-a-MUA rules (RDNS, Outlook, etc.) are happily
applied to my dialup customers, which is consistent with RW's
statement above.
Royce
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by RW <rw...@googlemail.com>.
On Fri, 9 Apr 2010 10:09:35 +0300
Henrik K <he...@hege.li> wrote:
> On Thu, Apr 08, 2010 at 10:26:27PM -0800, Royce Williams wrote:
> > >
> > Maybe I'm having a vocabulary problem. My MSAs are really also
> > MTAs - they receive mail from the customer, do an MX lookup on the
> > destination domain, and relay. But they are not MXes in that they
> > do not receive mail from foreign MTAs.
>
> Read and re-read "msa_networks" documentation. IMHO it's very clearly
> defined. It's just an extender for *_networks.
>
I think he may have put his finger on the problem in a previous post.
msa_networks defines the MSA by IP address. If SA runs on an MSA its
address is unlikely to be in the received headers. In that case SA has
no way of distinguishing an MSA from an MX server.
I would think that in this case the dynamic address blocks would need to
be explicitly defined.
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Henrik K <he...@hege.li>.
On Thu, Apr 08, 2010 at 10:26:27PM -0800, Royce Williams wrote:
> >
> > It also states that msa_networks propagates those hosts *_networks settings
> > recursively. Which means the dial-ups will be internal too.
>
> Ah, interesting. So I should explicitly *not* put my dialup MSAs in
> msa_networks, and put them only in trusted_networks.
Again, rules look for first external (non-internal) relay. Your suggestion
above does not make the dial-ups internal.
> Maybe I'm having a vocabulary problem. My MSAs are really also MTAs -
> they receive mail from the customer, do an MX lookup on the
> destination domain, and relay. But they are not MXes in that they do
> not receive mail from foreign MTAs.
Read and re-read "msa_networks" documentation. IMHO it's very clearly
defined. It's just an extender for *_networks.
"MSA means that the relay hosts on these networks accept mail from your own
users and authenticates them appropriately. These relays will never accept
mail from hosts that aren't authenticated in some way. Examples of
authentication include, IP lists, SMTP AUTH, POP-before-SMTP, etc."
"All relays found in the message headers after the MSA relay will take on
the same trusted and internal classifications as the MSA relay itself, as
defined by your trusted_networks and internal_networks configuration."
"Never include an MSA that also acts as an MX (or is also an intermediate
relay for an MX) or otherwise accepts mail from non-authenticated users in
msa_networks. Doing so will result in unknown external relays being
trusted."
So does your MSA accept mail only from your dial-up users or not? If that's
the case, I don't see what's the problem here.
> So maybe what I'm hearing is (thinking out loud):
>
> If I put my for-dialup MSAs in both msa_networks and internal_networks:
>
> * Everything that is in internal_networks must be included in trusted
> networks, per the Conf manpage.
> * Because of msa_networks propagation, my dialups become trusted to
> insert headers (bad).
Forget the trusted headers thing, I can't think of anything that it would
make "bad" in this scenario.
This is the configuration you want.
> If I put my for-dialup MSAs only in msa_networks:
>
> * My MSAs are seen as external.
> * My dialups gets penalized for non-content characteristics (coming
> from Outlook, bad HELOs, etc.) (bad)
Is this even possible?
> If I put my for-dialup MSAs only in trusted_networks:
>
> * My for-dialup MSAs are seen as external.
> * My dialups are seen as external and therefore penalized for
> non-content characteristics (bad).
Your dialup MSAs aren't external. Makes no sense.
> If I put my for-dialup MSAs both in trusted_networks and
> internal_networks, but not msa_networks:
>
> * My dialups aren't external, so they don't get spanked for being
> Outlook (good).
> * My dialups aren't trusted, so their headers are not trusted (good).
You wanted dial-ups to be internal. Makes no sense.
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Royce Williams <ro...@gmail.com>.
On Thu, Apr 8, 2010 at 8:25 PM, Henrik K <he...@hege.li> wrote:
> On Thu, Apr 08, 2010 at 06:31:37PM -0800, Royce Williams wrote:
>> On Thu, Apr 8, 2010 at 5:13 PM, Henrik K <he...@hege.li> wrote:
>> > On Thu, Apr 08, 2010 at 04:52:00PM -0800, Royce Williams wrote:
>> >>
>> >> Answering myself, I have reworked our *_networks to reflect our
>> >> architecture based on my re-re-re-reading. Nobody has said that my
>> >> example was broken (or was any good, for that matter), so I'm
>> >> operating from that.
>> >>
>> >> With all possible interfaces included from my dedicate MSAs in
>> >> msa_networks, my customers are still subject to IMG_DIRECT_TO_MX,
>> >> FSL_HELO_NON_FQDN_1, RDNS_NONE, HELO_NO_DOMAIN, DOS_DIRECT_TO_MX,
>> >> HELO_LOCALHOST, and the other "you look like an end user, not an MTA"
>> >> rules.
>> >>
>> >> Either my example is fundamentally broken, or everybody else is
>> >> already in there ripping and gripping rules anyway, and so don't mind
>> >> maintaining a similar list.
>> >>
>> >> Since there's no FAQ entry for this, but the reading for understanding
>> >> the problem is so dense, I'm starting to doubt my own sanity. :-)
>> >
>> > As said, these checks are made on the external border.
>> >
>> > Your example does not have MSAs defined as internal.
>>
>> By design. From the conf document:
>>
>> "Trusted relays that accept mail directly from dial-up connections
>> should not be listed in internal_networks. List them only in
>> trusted_networks."
>>
>> Is this incorrect?
>
> It also states that msa_networks propagates those hosts *_networks settings
> recursively. Which means the dial-ups will be internal too.
Ah, interesting. So I should explicitly *not* put my dialup MSAs in
msa_networks, and put them only in trusted_networks.
Maybe I'm having a vocabulary problem. My MSAs are really also MTAs -
they receive mail from the customer, do an MX lookup on the
destination domain, and relay. But they are not MXes in that they do
not receive mail from foreign MTAs.
So maybe what I'm hearing is (thinking out loud):
If I put my for-dialup MSAs in both msa_networks and internal_networks:
* Everything that is in internal_networks must be included in trusted
networks, per the Conf manpage.
* Because of msa_networks propagation, my dialups become trusted to
insert headers (bad).
If I put my for-dialup MSAs only in msa_networks:
* My MSAs are seen as external.
* My dialups gets penalized for non-content characteristics (coming
from Outlook, bad HELOs, etc.) (bad)
If I put my for-dialup MSAs only in trusted_networks:
* My for-dialup MSAs are seen as external.
* My dialups are seen as external and therefore penalized for
non-content characteristics (bad).
If I put my for-dialup MSAs both in trusted_networks and
internal_networks, but not msa_networks:
* My dialups aren't external, so they don't get spanked for being
Outlook (good).
* My dialups aren't trusted, so their headers are not trusted (good).
Is this correct?
Royce
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Henrik K <he...@hege.li>.
On Thu, Apr 08, 2010 at 06:31:37PM -0800, Royce Williams wrote:
> On Thu, Apr 8, 2010 at 5:13 PM, Henrik K <he...@hege.li> wrote:
> > On Thu, Apr 08, 2010 at 04:52:00PM -0800, Royce Williams wrote:
> >>
> >> Answering myself, I have reworked our *_networks to reflect our
> >> architecture based on my re-re-re-reading. Nobody has said that my
> >> example was broken (or was any good, for that matter), so I'm
> >> operating from that.
> >>
> >> With all possible interfaces included from my dedicate MSAs in
> >> msa_networks, my customers are still subject to IMG_DIRECT_TO_MX,
> >> FSL_HELO_NON_FQDN_1, RDNS_NONE, HELO_NO_DOMAIN, DOS_DIRECT_TO_MX,
> >> HELO_LOCALHOST, and the other "you look like an end user, not an MTA"
> >> rules.
> >>
> >> Either my example is fundamentally broken, or everybody else is
> >> already in there ripping and gripping rules anyway, and so don't mind
> >> maintaining a similar list.
> >>
> >> Since there's no FAQ entry for this, but the reading for understanding
> >> the problem is so dense, I'm starting to doubt my own sanity. :-)
> >
> > As said, these checks are made on the external border.
> >
> > Your example does not have MSAs defined as internal.
>
> By design. From the conf document:
>
> "Trusted relays that accept mail directly from dial-up connections
> should not be listed in internal_networks. List them only in
> trusted_networks."
>
> Is this incorrect?
It also states that msa_networks propagates those hosts *_networks settings
recursively. Which means the dial-ups will be internal too.
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Royce Williams <ro...@gmail.com>.
On Thu, Apr 8, 2010 at 5:13 PM, Henrik K <he...@hege.li> wrote:
> On Thu, Apr 08, 2010 at 04:52:00PM -0800, Royce Williams wrote:
>>
>> Answering myself, I have reworked our *_networks to reflect our
>> architecture based on my re-re-re-reading. Nobody has said that my
>> example was broken (or was any good, for that matter), so I'm
>> operating from that.
>>
>> With all possible interfaces included from my dedicate MSAs in
>> msa_networks, my customers are still subject to IMG_DIRECT_TO_MX,
>> FSL_HELO_NON_FQDN_1, RDNS_NONE, HELO_NO_DOMAIN, DOS_DIRECT_TO_MX,
>> HELO_LOCALHOST, and the other "you look like an end user, not an MTA"
>> rules.
>>
>> Either my example is fundamentally broken, or everybody else is
>> already in there ripping and gripping rules anyway, and so don't mind
>> maintaining a similar list.
>>
>> Since there's no FAQ entry for this, but the reading for understanding
>> the problem is so dense, I'm starting to doubt my own sanity. :-)
>
> As said, these checks are made on the external border.
>
> Your example does not have MSAs defined as internal.
By design. From the conf document:
"Trusted relays that accept mail directly from dial-up connections
should not be listed in internal_networks. List them only in
trusted_networks."
Is this incorrect?
Royce
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Henrik K <he...@hege.li>.
On Thu, Apr 08, 2010 at 04:52:00PM -0800, Royce Williams wrote:
>
> Answering myself, I have reworked our *_networks to reflect our
> architecture based on my re-re-re-reading. Nobody has said that my
> example was broken (or was any good, for that matter), so I'm
> operating from that.
>
> With all possible interfaces included from my dedicate MSAs in
> msa_networks, my customers are still subject to IMG_DIRECT_TO_MX,
> FSL_HELO_NON_FQDN_1, RDNS_NONE, HELO_NO_DOMAIN, DOS_DIRECT_TO_MX,
> HELO_LOCALHOST, and the other "you look like an end user, not an MTA"
> rules.
>
> Either my example is fundamentally broken, or everybody else is
> already in there ripping and gripping rules anyway, and so don't mind
> maintaining a similar list.
>
> Since there's no FAQ entry for this, but the reading for understanding
> the problem is so dense, I'm starting to doubt my own sanity. :-)
As said, these checks are made on the external border.
Your example does not have MSAs defined as internal.
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Royce Williams <ro...@gmail.com>.
On Wed, Apr 7, 2010 at 8:29 AM, Royce Williams <ro...@gmail.com> wrote:
> On Tue, Apr 6, 2010 at 2:13 PM, Kris Deugau <kd...@vianet.ca> wrote:
>> Royce Williams wrote:
>>>
>>> Some new information. In this 2008 thread:
>>>
>>> http://old.nabble.com/ALL_TRUSTED-and-DOS_OE_TO_MX-td15659736.html
>>>
>>> ... Daryl says:
>>>
>>> "So if (and I'll admit I don't think this occurred to me before) you're
>>> running SA on outgoing mail on your MSA right after you receive it (it's
>>> not relayed to an intermediate machine) SA can't detect the MSA and the
>>> whole msa_networks thing doesn't work."
>>>
>>> That is exactly our setup - our outbound servers are accepting mail
>>> from customers and handing them off to the world, not going through
>>> any other servers. Could this be the issue?
>>
>> Hmm. We have the same general setup, but we may be avoiding trouble because
>> our outbound scan is done while the SMTP transaction is in progress, and the
>> message SA sees does not have our MSA's Received: header yet. (Of course,
>> we then hit NO_RECEIVED and a collection of related tests, but none of them
>> score very high IIRC; have to check the specifics.)
>
> We also scan before accepting (MIMEDefang), so that must not be the
> difference between us.
>
> Can anyone else speak to whether or not Daryl's observation -- about
> msa_networks not applying to MSAs that are at the email "border" -- is
> still in effect?
Answering myself, I have reworked our *_networks to reflect our
architecture based on my re-re-re-reading. Nobody has said that my
example was broken (or was any good, for that matter), so I'm
operating from that.
With all possible interfaces included from my dedicate MSAs in
msa_networks, my customers are still subject to IMG_DIRECT_TO_MX,
FSL_HELO_NON_FQDN_1, RDNS_NONE, HELO_NO_DOMAIN, DOS_DIRECT_TO_MX,
HELO_LOCALHOST, and the other "you look like an end user, not an MTA"
rules.
Either my example is fundamentally broken, or everybody else is
already in there ripping and gripping rules anyway, and so don't mind
maintaining a similar list.
Since there's no FAQ entry for this, but the reading for understanding
the problem is so dense, I'm starting to doubt my own sanity. :-)
Royce
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Royce Williams <ro...@gmail.com>.
On Tue, Apr 6, 2010 at 2:13 PM, Kris Deugau <kd...@vianet.ca> wrote:
> Royce Williams wrote:
>>
>> Some new information. In this 2008 thread:
>>
>> http://old.nabble.com/ALL_TRUSTED-and-DOS_OE_TO_MX-td15659736.html
>>
>> ... Daryl says:
>>
>> "So if (and I'll admit I don't think this occurred to me before) you're
>> running SA on outgoing mail on your MSA right after you receive it (it's
>> not relayed to an intermediate machine) SA can't detect the MSA and the
>> whole msa_networks thing doesn't work."
>>
>> That is exactly our setup - our outbound servers are accepting mail
>> from customers and handing them off to the world, not going through
>> any other servers. Could this be the issue?
>
> Hmm. We have the same general setup, but we may be avoiding trouble because
> our outbound scan is done while the SMTP transaction is in progress, and the
> message SA sees does not have our MSA's Received: header yet. (Of course,
> we then hit NO_RECEIVED and a collection of related tests, but none of them
> score very high IIRC; have to check the specifics.)
We also scan before accepting (MIMEDefang), so that must not be the
difference between us.
Can anyone else speak to whether or not Daryl's observation -- about
msa_networks not applying to MSAs that are at the email "border" -- is
still in effect?
Royce
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Kris Deugau <kd...@vianet.ca>.
Royce Williams wrote:
> Some new information. In this 2008 thread:
>
> http://old.nabble.com/ALL_TRUSTED-and-DOS_OE_TO_MX-td15659736.html
>
> ... Daryl says:
>
> "So if (and I'll admit I don't think this occurred to me before) you're
> running SA on outgoing mail on your MSA right after you receive it (it's
> not relayed to an intermediate machine) SA can't detect the MSA and the
> whole msa_networks thing doesn't work."
>
> That is exactly our setup - our outbound servers are accepting mail
> from customers and handing them off to the world, not going through
> any other servers. Could this be the issue?
Hmm. We have the same general setup, but we may be avoiding trouble
because our outbound scan is done while the SMTP transaction is in
progress, and the message SA sees does not have our MSA's Received:
header yet. (Of course, we then hit NO_RECEIVED and a collection of
related tests, but none of them score very high IIRC; have to check the
specifics.)
-kgd
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Royce Williams <ro...@gmail.com>.
On Mon, Apr 5, 2010 at 11:10 AM, Kris Deugau <kd...@vianet.ca> wrote:
> Royce Williams wrote:
>>
>> What is the optimal configuration (local.cf or other) for an ISP's
>> MSAs to prevent unauthenticated dynamic-IP customers from triggering
>> dynamic tests, but still benefiting from general filtering?
>>
>> I was hoping for a magical 'mua_networks' option, which let me
>> enumerate the IP space that my users submit from, and automatically
>> exempt them from DOS_OE_TO_MX, etc., but I haven't been able to find
>> anything like that.
>>
>> From my reading of the .conf manpage, the TrustPath page, and the
>> archives (see references below), I've tentatively concluded that I
>> will need to have some local rule overrides on all of my MSAs for any
>> rule or meta-rule that detects dynamic-looking hostnames ... but that
>> seems high-maintenance locally as well as a lot of duplicated work for
>> other SA users.
>
> Read, read, and re-read. It's a bit tangled and confusing balancing the
> various requirements but you should be able to get it right with a little
> effort.
>
> To summarize what I've applied here:
>
> trusted_networks: Contains CIDR ranges for our servers. These systems are
> "trusted" in that we know they will not forge Received: headers. I've added
> a number of third-party systems here for various reasons.
>
> internal_networks: IPs or CIDR ranges for your inbound mail flow, *within
> your network*. Usually equivalent to trusted_networks, but not always;
> must be *entirely contained by* trusted_networks. I've included one of
> Postini's IP ranges here to catch mail relayed to domains handled by Postini
> that might otherwise have been blocked at the MTA level by Spamhaus' Zen.
>
> msa_networks: IPs of CIDR ranges for your outbound mail flow, IE systems
> that accept mail from your authorized customer IP ranges. (and anywhere
> else via SMTP AUTH or similar). As far as I can tell, these *may* overlap
> internal_networks but if you're big enough that these settings are a
> problem, they probably don't. Also a subset of trusted_networks. (FWIW, I
> found overlapping this with internal_networks caused problems. YMMV.)
>
> We scan all outbound mail with the same SA cluster as our inbound scanning,
> and I haven't seen misbehaviour I could blame on these settings since
> sometime last summer (couple of corner-case oddities IIRC); before that it
> would have been more than a year since I dug into them in detail and added
> the msa_networks entries along with the upgrade to SA 3.2 (IIRC).
Kris, thanks for the good summary.
Some new information. In this 2008 thread:
http://old.nabble.com/ALL_TRUSTED-and-DOS_OE_TO_MX-td15659736.html
... Daryl says:
"So if (and I'll admit I don't think this occurred to me before) you're
running SA on outgoing mail on your MSA right after you receive it (it's
not relayed to an intermediate machine) SA can't detect the MSA and the
whole msa_networks thing doesn't work."
That is exactly our setup - our outbound servers are accepting mail
from customers and handing them off to the world, not going through
any other servers. Could this be the issue?
Also, I think that an example snippet of.cf illustrating and briefly
explaining each of the three _networks options might be in order, and
might make the reading, re-reading, and re-reading of the docs a
little less painful. Writing one will also demonstrate that I've
correctly grokked what's been going on here. :-) I'll take a stab at
one.
Royce
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Kris Deugau <kd...@vianet.ca>.
Royce Williams wrote:
> What is the optimal configuration (local.cf or other) for an ISP's
> MSAs to prevent unauthenticated dynamic-IP customers from triggering
> dynamic tests, but still benefiting from general filtering?
>
> I was hoping for a magical 'mua_networks' option, which let me
> enumerate the IP space that my users submit from, and automatically
> exempt them from DOS_OE_TO_MX, etc., but I haven't been able to find
> anything like that.
>
> From my reading of the .conf manpage, the TrustPath page, and the
> archives (see references below), I've tentatively concluded that I
> will need to have some local rule overrides on all of my MSAs for any
> rule or meta-rule that detects dynamic-looking hostnames ... but that
> seems high-maintenance locally as well as a lot of duplicated work for
> other SA users.
Read, read, and re-read. It's a bit tangled and confusing balancing the
various requirements but you should be able to get it right with a
little effort.
To summarize what I've applied here:
trusted_networks: Contains CIDR ranges for our servers. These systems
are "trusted" in that we know they will not forge Received: headers.
I've added a number of third-party systems here for various reasons.
internal_networks: IPs or CIDR ranges for your inbound mail flow,
*within your network*. Usually equivalent to trusted_networks, but not
always; must be *entirely contained by* trusted_networks. I've
included one of Postini's IP ranges here to catch mail relayed to
domains handled by Postini that might otherwise have been blocked at the
MTA level by Spamhaus' Zen.
msa_networks: IPs of CIDR ranges for your outbound mail flow, IE
systems that accept mail from your authorized customer IP ranges. (and
anywhere else via SMTP AUTH or similar). As far as I can tell, these
*may* overlap internal_networks but if you're big enough that these
settings are a problem, they probably don't. Also a subset of
trusted_networks. (FWIW, I found overlapping this with
internal_networks caused problems. YMMV.)
We scan all outbound mail with the same SA cluster as our inbound
scanning, and I haven't seen misbehaviour I could blame on these
settings since sometime last summer (couple of corner-case oddities
IIRC); before that it would have been more than a year since I dug into
them in detail and added the msa_networks entries along with the upgrade
to SA 3.2 (IIRC).
-kgd
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by John Hardin <jh...@impsec.org>.
On Sat, 3 Apr 2010, Henrik K wrote:
> On Fri, Apr 02, 2010 at 01:45:57PM -0800, Royce Williams wrote:
>> What is the optimal configuration (local.cf or other) for an ISP's
>> MSAs to prevent unauthenticated dynamic-IP customers from triggering
>> dynamic tests, but still benefiting from general filtering?
>>
>> I was hoping for a magical 'mua_networks' option, which let me
>> enumerate the IP space that my users submit from, and automatically
>> exempt them from DOS_OE_TO_MX, etc., but I haven't been able to find
>> anything like that.
>
> All dynamic rules look at external relays. So if you have SA on the relay
> that accepts mail from dynamic space, you need to include all that in
> internal_networks and disable ALL_TRUSTED since it would always hit. I think
> only other option is to manually disable all affected rules, which would be
> hard to maintain..
Or add a custom rule that checks the last external relay for the IP spaces
you trust and add some negative points.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Look at the people at the top of both efforts. Linus Torvalds is a
university graduate with a CS degree. Bill Gates is a university
dropout who bragged about dumpster-diving and using other peoples'
garbage code as the basis for his code. Maybe that has something to
do with the difference in quality/security between Linux and
Windows. -- anytwofiveelevenis on Y! SCOX
-----------------------------------------------------------------------
10 days until Thomas Jefferson's 267th Birthday
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by RW <rw...@googlemail.com>.
On Sat, 3 Apr 2010 10:24:43 -0800
Royce Williams <ro...@gmail.com> wrote:
> > Putting the address ranges into internal_networks is what you do if
> > you *don't* have separate MSAs and MX servers. Otherwise you you
> > put the MSAs into msa_networks and internal_networks. Anything that
> > connects to a server in msa_networks inherits the internal/trusted
> > status of the msa.
> >
> My understanding is that if my own dynamics inherit the MSA's
> internal/trusted status, then the headers added by those hosts are
> assumed to be genuine. That's a behavior I'm trying to avoid.
That's the behaviour you want. If everything is internal, there's
nothing to run the tests against.
> Maybe I'm misunderstanding some rule fundamentals. Some rules are
> designed to detect MUAs, but don't appear to be affected by the
> contents of msa_networks. Examples are the Outlook-detecting rules
> like DOS_OE_TO_MX, ...
DOS_OE_TO_MX depends on __DOS_SINGLE_EXT_RELAY which is
looking for one "[...]" block in Relays-External, which should be
empty.
Do you have your MSAs in msa_networks, internal_networks and
trusted_networks? If you do, and you have a non-null Relays-External,
you've probably found a bug.
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Royce Williams <ro...@gmail.com>.
Whoops - forgot to reply-all; resending with minor modifications.
On Sat, Apr 3, 2010 at 9:10 AM, RW <rw...@googlemail.com> wrote:
> On Sat, 3 Apr 2010 06:18:25 -0800
> Royce Williams <ro...@gmail.com> wrote:
>
>> On Fri, Apr 2, 2010 at 11:20 PM, Henrik K <he...@hege.li> wrote:
>> > On Fri, Apr 02, 2010 at 01:45:57PM -0800, Royce Williams wrote:
>> >> What is the optimal configuration (local.cf or other) for an ISP's
>> >> MSAs to prevent unauthenticated dynamic-IP customers from
>> >> triggering dynamic tests, but still benefiting from general
>> >> filtering?
>> >>
>> >> I was hoping for a magical 'mua_networks' option, which let me
>> >> enumerate the IP space that my users submit from, and automatically
>> >> exempt them from DOS_OE_TO_MX, etc., but I haven't been able to
>> >> find anything like that.
>> >
>> > All dynamic rules look at external relays. So if you have SA on the
>> > relay that accepts mail from dynamic space, you need to include all
>> > that in internal_networks and disable ALL_TRUSTED since it would
>> > always hit.
>>
>> Interesting. I do have SA on my MSAs. That might be the single knob
>> that I am looking for.
>>
>> Some folks aren't big enough to have separated MTAs and MSAs; for
>> their benefit, is there any other approach that would work? My
>> imagined mua_networks would fit this bill, and it would allow the same
>> SA configuration on both MTAs and MSAs, which would make my life as a
>> sysadmin easier.
>
> Putting the address ranges into internal_networks is what you do if you
> *don't* have separate MSAs and MX servers. Otherwise you you put the
> MSAs into msa_networks and internal_networks. Anything that connects to
> a server in msa_networks inherits the internal/trusted status of the
> msa.
>
> Even if you don't or can't do any of the above, SA will pick-up and on
> authentication recorded in the received header and for the most part do
> the right thing, e.g. it won't run PBL/DUL lookups.
My understanding is that if my own dynamics inherit the MSA's
internal/trusted status, then the headers added by those hosts are
assumed to be genuine. That's a behavior I'm trying to avoid.
Maybe I'm misunderstanding some rule fundamentals. Some rules are
designed to detect MUAs, but don't appear to be affected by the
contents of msa_networks. Examples are the Outlook-detecting rules
like DOS_OE_TO_MX, some of the HELO-enforcement rules, and the
letter-number-letter hostname rules. From my testing, even though all
of my MSAs have msa_networks fully populated, these rules are applied
to all clients using my MSAs.
If I enumerate and disable all of the MUA-spanking rules, I get the
whack-a-mole problem that I have today. If I upgrade SA or update to
get new rulesets, new MUA spanking-rules may appear. I then have to
track down new false positives from my customer base. It is difficult
to test the rule changes in advance, because of the sheer number of
MUA families and versions.
I also considered John Hardin's idea about adding negative points,
which would work, but is not precise enough for me. I want my
infected customers to be spanked for spam as much as possible -- but
not for being clients. Picking an arbitrary negative number will
result in more customer spam getting out.
In other words, I want to use SA to express that my dynamics:
* are allowed to submit
* should not be trusted to insert only good headers
* should not be punished for being MUAs
* should not be punished for being dynamic
* should be subject to all other rules
Is there a way to express this?
Royce
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by RW <rw...@googlemail.com>.
On Sat, 3 Apr 2010 06:18:25 -0800
Royce Williams <ro...@gmail.com> wrote:
> On Fri, Apr 2, 2010 at 11:20 PM, Henrik K <he...@hege.li> wrote:
> > On Fri, Apr 02, 2010 at 01:45:57PM -0800, Royce Williams wrote:
> >> What is the optimal configuration (local.cf or other) for an ISP's
> >> MSAs to prevent unauthenticated dynamic-IP customers from
> >> triggering dynamic tests, but still benefiting from general
> >> filtering?
> >>
> >> I was hoping for a magical 'mua_networks' option, which let me
> >> enumerate the IP space that my users submit from, and automatically
> >> exempt them from DOS_OE_TO_MX, etc., but I haven't been able to
> >> find anything like that.
> >
> > All dynamic rules look at external relays. So if you have SA on the
> > relay that accepts mail from dynamic space, you need to include all
> > that in internal_networks and disable ALL_TRUSTED since it would
> > always hit.
>
> Interesting. I do have SA on my MSAs. That might be the single knob
> that I am looking for.
>
> Some folks aren't big enough to have separated MTAs and MSAs; for
> their benefit, is there any other approach that would work? My
> imagined mua_networks would fit this bill, and it would allow the same
> SA configuration on both MTAs and MSAs, which would make my life as a
> sysadmin easier.
Putting the address ranges into internal_networks is what you do if you
*don't* have separate MSAs and MX servers. Otherwise you you put the
MSAs into msa_networks and internal_networks. Anything that connects to
a server in msa_networks inherits the internal/trusted status of the
msa.
Even if you don't or can't do any of the above, SA will pick-up and on
authentication recorded in the received header and for the most part do
the right thing, e.g. it won't run PBL/DUL lookups.
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Royce Williams <ro...@gmail.com>.
On Fri, Apr 2, 2010 at 11:20 PM, Henrik K <he...@hege.li> wrote:
> On Fri, Apr 02, 2010 at 01:45:57PM -0800, Royce Williams wrote:
>> What is the optimal configuration (local.cf or other) for an ISP's
>> MSAs to prevent unauthenticated dynamic-IP customers from triggering
>> dynamic tests, but still benefiting from general filtering?
>>
>> I was hoping for a magical 'mua_networks' option, which let me
>> enumerate the IP space that my users submit from, and automatically
>> exempt them from DOS_OE_TO_MX, etc., but I haven't been able to find
>> anything like that.
>
> All dynamic rules look at external relays. So if you have SA on the relay
> that accepts mail from dynamic space, you need to include all that in
> internal_networks and disable ALL_TRUSTED since it would always hit.
Interesting. I do have SA on my MSAs. That might be the single knob
that I am looking for.
Some folks aren't big enough to have separated MTAs and MSAs; for
their benefit, is there any other approach that would work? My
imagined mua_networks would fit this bill, and it would allow the same
SA configuration on both MTAs and MSAs, which would make my life as a
sysadmin easier.
Is everyone else solving it this way - by populating internal_networks
on their MSAs with their dynamic space, combined with disabling
ALL_TRUSTED? I'm trying to get a sense of 'best practice' here.
Royce
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Alex <my...@gmail.com>.
Hi,
>> What is the optimal configuration (local.cf or other) for an ISP's
>> MSAs to prevent unauthenticated dynamic-IP customers from triggering
>> dynamic tests, but still benefiting from general filtering?
>>
>> I was hoping for a magical 'mua_networks' option, which let me
>> enumerate the IP space that my users submit from, and automatically
>> exempt them from DOS_OE_TO_MX, etc., but I haven't been able to find
>> anything like that.
I've added ranges of networks to $mynetworks in postfix, effectively
making them a trusted relay, but their IPs still shift quite
frequently of course, so it's not a very good solution.
Maybe an option would be to create another postfix instance to skip
scanning altogether based on pop-before-smtp auth or something?
I don't believe TLS would help here either, right? It would be nice,
but it only supports encryption and not authentication on Outlook
clients, right?
The real solution to this is smtp-auth, which is the way Google does it, right?
> All dynamic rules look at external relays. So if you have SA on the relay
> that accepts mail from dynamic space, you need to include all that in
> internal_networks and disable ALL_TRUSTED since it would always hit.
Yes, and due to the nature of dynamic being dynamic, it's a real
challenge. Any suggestions on how to populate that with trusted
information, then?
Thanks,
Alex
Re: skipping dynamic tests for ISP's own dynamic networks?
Posted by Henrik K <he...@hege.li>.
On Fri, Apr 02, 2010 at 01:45:57PM -0800, Royce Williams wrote:
> What is the optimal configuration (local.cf or other) for an ISP's
> MSAs to prevent unauthenticated dynamic-IP customers from triggering
> dynamic tests, but still benefiting from general filtering?
>
> I was hoping for a magical 'mua_networks' option, which let me
> enumerate the IP space that my users submit from, and automatically
> exempt them from DOS_OE_TO_MX, etc., but I haven't been able to find
> anything like that.
All dynamic rules look at external relays. So if you have SA on the relay
that accepts mail from dynamic space, you need to include all that in
internal_networks and disable ALL_TRUSTED since it would always hit. I think
only other option is to manually disable all affected rules, which would be
hard to maintain..