You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by bu...@bugzilla.spamassassin.org on 2007/03/08 23:24:19 UTC
[Bug 5374] New: DUL/Dynablock RBL test issues
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5374
Summary: DUL/Dynablock RBL test issues
Product: Spamassassin
Version: 3.1.8
Platform: All
OS/Version: All
Status: NEW
Severity: normal
Priority: P5
Component: Rules (Eval Tests)
AssignedTo: dev@spamassassin.apache.org
ReportedBy: apache@robm.fastmail.fm
Consider some common mail routes.
* Common good mail routes
user ip -> isp/3rd party smtp -> receiver
user ip -> isp/3rd party smtp -> receiver (forwards to) -> receiver
* Rarer goodmail routes
internal ip -> sme mail router (on dsl connection) -> receiver
* Spam routes
spammer user ip -> isp/3rd party smtp -> receiver
zombie user ip -> receiver
zombie user ip -> forwarder -> receiver
zombie user ip -> webmail -> receiver (new spam route!)
And the RBLs seem to fall into two main types.
* sbl - verified spam sources and spam operations
- We want to check that against the true entry point, which is the first
"untrusted" IP, whatever it is
* xbl - illegal 3rd party exploits, including open proxies, worms/viruses with
built-in spam engines, and other types of trojan-horse exploits.
- We want to check that against the true entry point, which is the first
"untrusted" IP, whatever it is
* pbl - end-user IP address ranges which should not be delivering
unauthenticated SMTP email to any Internet mail server except those provided for
specifically by an ISP
- We don't want to check the final IP against this, because we actually expect
it to be in the PBL
* njabl-dul - includes dynamic assigned ip-addresses (DHCP) of ADSL and Cable
which stops even unknown trojan cold
- We don't want to check the final IP against this, because we actually expect
it to be in the njabl-dul
So for things like the sbl/xbl, we want to trust the isp/3rd party servers to
get back to the actual user ip. This is mostly doable (see
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5373 for a suggestion on
how to do this more effectively in the forwarding case)
For the pbl/njabl-dul, we have a problem. How do we tell the difference between:
user ip -> isp/3rd party smtp -> receiver
zombie user ip -> forwarder -> receiver
There may be information in the Received headers that it was an authenticated
SMTP session, but that can't be guaranteed, so I think that you can't use dialup
RBLs in these cases, it's impossible to accurately tell the difference.
Also if you try and use a dialup RBL in this case:
zombie user ip -> receiver
It also breaks this:
internal ip -> sme mail router (on adsl connection) -> receiver
And if you have internally delivered email, it might break this case as well:
user ip -> 3rd party auth smtp -> same 3rd party recipient
In that case, if your mail server doesn't add something to the header to note
it's an authenticated SMTP session, we'd lookup the DUL against the user ip,
even though they used auth smtp.
Unfortunately I don't think there's an easy solution that allows the use of
DUL/Dynablock type RBLs consistently and accurately. I think DUL type blocklists
should only be used if:
1. You only check against the first noninternal IP (quite possibly different to
first nontrusted IP if you have a bigger trust algorithm like the one from BUG 5373)
2. If you accept SMTP auth mail for local users, your mail server does add the
appropriate Received header that parse_received_line can detect as an
authenticated SMTP session
3. You're willing to penalise SME type customers that run their own mail servers
on DSL/dialup lines
I think there should be a pretty much global switch to enable/disable these
dynablock/dul type lists that people can change if they are sure their system
conforms to the above requirements
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 5374] DUL/Dynablock RBL test issues
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5374
jm@jmason.org changed:
What |Removed |Added
----------------------------------------------------------------------------
Target Milestone|Undefined |3.3.0
------- Additional Comments From jm@jmason.org 2007-03-13 08:40 -------
also useful to contemplate for 3.3.0
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 5374] DUL/Dynablock RBL test issues
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5374
------- Additional Comments From kai-sa-bugs@spamshield.org 2007-03-19 07:59 -------
(In reply to comment #0)
> * pbl - end-user IP address ranges which should not be delivering
> unauthenticated SMTP email to any Internet mail server except those provided for
> specifically by an ISP
> - We don't want to check the final IP against this, because we actually expect
> it to be in the PBL
note that "authenticated" also means "authorized to relay by MTA", e.g.:
customers sitting in the ISP's own IP space that are allowed to relay without
authentication by default, whether they use SMTP-Auth or not.
>
> * njabl-dul - includes dynamic assigned ip-addresses (DHCP) of ADSL and Cable
> which stops even unknown trojan cold
> - We don't want to check the final IP against this, because we actually expect
> it to be in the njabl-dul
With the launch of PBL, the "dynablock.njabl.org" zone (127.0.0.3 code),
which is also present in the combined.njabl.org zone (127.0.0.3 code)
has become one and the same as the PBL.
NJABL-DUL in this context was the former "dynablock.easynet.nl" zone, that
evolved substantially since the original DUL shut down.
> For the pbl/njabl-dul, we have a problem. How do we tell the difference between:
>
> user ip -> isp/3rd party smtp -> receiver
> zombie user ip -> forwarder -> receiver
You can't. Webhoster 'mail forwarder' type of MTAs are probably the single
worst problem for large email receivers (ISPs) today:
They are often poorly or not at all spam-filtered (often with intent, and with the
excuse of "our customers have the right to receive all mail sent to
their email addresses"!), with spam:ham ratios reaching 90% (at least
reasonably filtered forwarders reach 50-60%) - basically offloading
spamfiltering to the receiving MTAs where it hurts most: post-mail-acceptance
content scanning (CPU intensive).
And yet, unless you are a SORBS-using moron of a MTA-admin, you will not
deny these forwarder's traffic at the front door, because your own
customers have hosted (and mail-forwarded) accounts there.
>
> There may be information in the Received headers that it was an authenticated
> SMTP session, but that can't be guaranteed, so I think that you can't use dialup
> RBLs in these cases, it's impossible to accurately tell the difference.
Both SMTP-Auth'd and relay-permitted-by-IP situations exist concurrently,
even on the same MTA, so you can't tell the difference.
Do not attempt to use the PBL for this purpose.
It'll even be troublesome to use XBL for behind-MTA Received header checks,
with a noticeable FP rate, but I would suggest further testing for this,
as it'll probably hit the guilty people and poorly managed ISPs (and their
poorly managed, non-SMTP-filtered dynamic ranges) quite disproportionally.
>
> Also if you try and use a dialup RBL in this case:
>
> zombie user ip -> receiver
>
> It also breaks this:
>
> internal ip -> sme mail router (on adsl connection) -> receiver
>
> And if you have internally delivered email, it might break this case as well:
>
> user ip -> 3rd party auth smtp -> same 3rd party recipient
>
> In that case, if your mail server doesn't add something to the header to note
> it's an authenticated SMTP session, we'd lookup the DUL against the user ip,
> even though they used auth smtp.
>
> Unfortunately I don't think there's an easy solution that allows the use of
> DUL/Dynablock type RBLs consistently and accurately. I think DUL type blocklists
> should only be used if:
>
> 1. You only check against the first noninternal IP (quite possibly different to
> first nontrusted IP if you have a bigger trust algorithm like the one from BUG
5373)
> 2. If you accept SMTP auth mail for local users, your mail server does add the
> appropriate Received header that parse_received_line can detect as an
> authenticated SMTP session
> 3. You're willing to penalise SME type customers that run their own mail servers
> on DSL/dialup lines
>
> I think there should be a pretty much global switch to enable/disable these
> dynablock/dul type lists that people can change if they are sure their system
> conforms to the above requirements
ISPs (and other institutions) must not use DNSBLs to deny service to their
own customers - that's particularly true for their own IP space, but also
customers sending mail via SMTP-Auth+Submit from other locations.
Whether they use DNSBLs to flag accounts or rate-limit them, is up to then,
but it has to work by default.
The only clean solution is to separate incoming MTAs (SMTP from world) from
outgoing/relaying MTAs (SMTP-Auth+Submit or permit local IP space), and
apply different SA rulesets on both.
On combined servers, this can get VERY messy - and SA admins most certainly
will have to make their own IP space used by their customers "trusted",
and with it disable all DNSBL rules against those IPs.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 5374] DUL/Dynablock RBL test issues
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5374
------- Additional Comments From apache@robm.fastmail.fm 2007-03-19 19:15 -------
> And yet, unless you are a SORBS-using moron of a MTA-admin, you will not
> deny these forwarder's traffic at the front door, because your own
> customers have hosted (and mail-forwarded) accounts there.
Exactly. See:
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5373
For some suggestions on helping with this case with other RBLs as well.
I didn't really see any comment about my main assertion which is:
> > I think there should be a pretty much global switch to enable/disable these
> > dynablock/dul type lists that people can change if they are sure their system
> > conforms to the above requirements
Basically as I mentioned, there's a number of preconditions your system must
have before using any of the DUL type RBLs (see points 1-3 in my original post)
and I think a lot of systems wouldn't meet those preconditions, so I think
having some way to easily disable all DUL type RBLs is a good idea, and probably
should be the case by default?
Rob
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 5374] DUL/Dynablock RBL test issues
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5374
------- Additional Comments From jm@jmason.org 2007-03-20 08:17 -------
I'd like to hear Daryl's comments, seeing as he's become the
trusted_networks/internal_networks guru ;)
> I think DUL type blocklists should only be used if:
>
> 1. You only check against the first noninternal IP (quite possibly different
> to first nontrusted IP if you have a bigger trust algorithm like the one from
> BUG 5373)
Are you taking the differentiation between trusted_networks and
internal_networks into account? I noticed an issue on bug 5373 (at least in
terminology) on that point.
internal_networks was designed to deal with this issue, since it *does* allow
trust to extend further, without affecting the network boundary used to
determine which IP to check in the DUL case. The DUL/PBL rules should all be
using "-lastexternal" accordingly to take that into account.
We definitely do not look further than the "handover to receiver" IP
for DUL rules, to avoid this case:
> user ip -> isp/3rd party smtp -> receiver
> zombie user ip -> forwarder -> receiver
That limitation does miss some spam for the forwarder case, but that's better
than increasing FPs in the case of "user ip -> 3rd party smtp server in
trusted_networks -> receiver".
> 2. If you accept SMTP auth mail for local users, your mail server does add the
> appropriate Received header that parse_received_line can detect as an
> authenticated SMTP session
Yep -- documented as an important thing that ISPs need to do.
> 3. You're willing to penalise SME type customers that run their own mail servers
> on DSL/dialup lines
We haven't made a definite stand on this, but I think those guys are pretty
much a lost cause by now. :( SpamAssassin is the _least_ of their worries,
with pretty much every major ISP (including AOL) blocking them, afaik.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 5374] DUL/Dynablock RBL test issues
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5374
------- Additional Comments From jm@jmason.org 2007-03-22 04:01 -------
thanks for commenting, Daryl!
(In reply to comment #5)
> Also note, and this could have a sizable affect, there's a whole pile of rules
> that use X-Spam-Relays-Trusted/Untrusted where they should really use
> X-Spam-Relays-Internal/External. Everyone seems to write rules that totally
> ignores the possibility of someone extending only trusted_networks towards the
> sender.
agreed. We need to (a) note this in the documentation (now done) and (b) fix
those rules if they appear in our rulesets (separate issue).
> In any case... this is really more oriented towards a discussion on one of the
> mailing lists rather than in bugzilla.
heh. actually, I asked Rob to post it here ;)
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 5374] DUL/Dynablock RBL test issues
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5374
------- Additional Comments From spamassassin@dostech.ca 2007-03-22 00:45 -------
Disclaimer: this *week* feels like the busiest *year* I've had in a long time.
In an effort to stop waiting until I have time to answer this in length I'll
supply my comments in terse & cranky mode. Apologies in advance for probably
seeming like an ass.
Summary mode: As far as I can tell the main point of this bug is to get support
for bug 5373 which you don't need support for, you've already got it. :)
(In reply to comment #0)
> * Rarer goodmail routes
> internal ip -> sme mail router (on dsl connection) -> receiver
This is no different than if they were using their ISP's MSA. If it's listed IP
space they can't use it, though, and need to smarthost through a non-listed
server. Just like you wouldn't use an ISP's MSA if they can't keep it out of
blacklists.
Of course if they've got DSL and it's not listed, they're free to use it. Good
ISPs will provide useful static IP space for DSL customers, not-so-good
providers will claim they do but don't. My business' ISP falls in the latter
category, I deal with it by smarthosting elsewhere. That's life.
> * Spam routes
> spammer user ip -> isp/3rd party smtp -> receiver
It's not clear who's ISP (the spammer's or the receiver's) that you are
referring to, but if it's the spammer's then DNSBL the ISP. If it's the
receiver's then the receiver should know who their ISP is and can configure
trust accordingly.
> zombie user ip -> receiver
> zombie user ip -> forwarder -> receiver
Yeah. Forwarding sucks, that's life. If you know who is forwarding mail for
your user's then you can setup trust accordingly. If you don't know who is
forwarding mail for your users then your accuracy is going to suffer.
I've got some thoughts on how to dynamically determine who is legitimately
forwarding mail for your users, but for inclusion in SA I need to get
Received.pm pluginized first.
> zombie user ip -> webmail -> receiver (new spam route!)
Not new at all. DNSBL support for this is already present and pretty much works
the same way it would absent a webmail hop.
[stuff about various DNSBLs]
> So for things like the sbl/xbl, we want to trust the isp/3rd party servers to
> get back to the actual user ip. This is mostly doable (see
> http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5373 for a suggestion on
> how to do this more effectively in the forwarding case)
bug 5373 doesn't actually make this more effective, it just makes it less work.
> For the pbl/njabl-dul, we have a problem. How do we tell the difference between:
>
> user ip -> isp/3rd party smtp -> receiver
> zombie user ip -> forwarder -> receiver
>
> There may be information in the Received headers that it was an authenticated
> SMTP session, but that can't be guaranteed, so I think that you can't use dialup
> RBLs in these cases, it's impossible to accurately tell the difference.
Again, I have no idea who's ISP/3rd part smtp you're talking about. I agree
though, if you can't tell who's servers are who's you can't properly configure
SA. If you know who handles your mail though, you should have no problem at all.
> Also if you try and use a dialup RBL in this case:
>
> zombie user ip -> receiver
>
> It also breaks this:
>
> internal ip -> sme mail router (on adsl connection) -> receiver
Only if the ADSL connection is in listed IP space, but of course -- don't do it.
> And if you have internally delivered email, it might break this case as well:
>
> user ip -> 3rd party auth smtp -> same 3rd party recipient
>
> In that case, if your mail server doesn't add something to the header to note
> it's an authenticated SMTP session, we'd lookup the DUL against the user ip,
> even though they used auth smtp.
SA supports pretty much anyway that this can happen and be determined by looking
at the headers. There's auth tokens, RFC3848 with types and a POP-before-SMTP
plugin. If one of those doesn't satisfy the problem then that 3rd part smtp
needs to use a milter/whatever that is smart enough not to scan mail from auth'd
clients. If somebody tries to scan the mail after the 3rd part smtp passes it
to them they've shouldn't trust that 3rd party MSA or they should use msa_networks.
Bottom line is, SA does everything possible to make this work with at least the
4 most common MTAs.
> Unfortunately I don't think there's an easy solution that allows the use of
> DUL/Dynablock type RBLs consistently and accurately. I think DUL type blocklists
> should only be used if:
>
> 1. You only check against the first noninternal IP (quite possibly different to
> first nontrusted IP if you have a bigger trust algorithm like the one from BUG
5373)
We already do this. bug 5373 doesn't introduce anything new/special to do with
this.
> 2. If you accept SMTP auth mail for local users, your mail server does add the
> appropriate Received header that parse_received_line can detect as an
> authenticated SMTP session
Or you've segreate your MSA from other MTA functions and can use msa_networks.
Or you can use the POPAuth plugin for POP-before-SMTP. Or you know your user's IPs.
With more and more ISPs blocking port 25 and more and more MTAs including auth
tokens in their recieved headers for port 587/465 submission clients this is
become more and more of a non-issue each day.
> 3. You're willing to penalise SME type customers that run their own mail servers
> on DSL/dialup lines
ABSOLUTELY and I'm willing to penalize any small to medium enterprise (public
companies with up to about $100 million in revenue) that are so cheap and silly
about their IT infrastucture.
SMBs are in the same boat. Although it's not a penalty, it's just reality. If
they look like any other spam spewing host on the 'net then there isn't a
rational being that can seriously argure that they shouldn't have to smarthost
their mail though a host that doesn't like just like any other spam spewing
host. Relaying mail through their ISP, or renting a $10 a month virtual private
server to do it if they're concerned about their ISP for whatever reason, solves
this problem.
20 years of literally living in small to huge businesses tells me that if you
can afford to do it right then this isn't an issue. If you can't afford to do
it right then you don't have the time, or the ability to bear any other cost
metric you can imagine, to do it the wrong way -- just smarthost the mail and be
done with it.
> I think there should be a pretty much global switch to enable/disable these
> dynablock/dul type lists that people can change if they are sure their system
> conforms to the above requirements
There is. skip_rbl_checks 1 and disable the SPF plugin if you want. If there's
something that isn't turned off that should be when skip_rbl_checks is enabled
we can address that.
(In reply to comment #3)
> Basically as I mentioned, there's a number of preconditions your system must
> have before using any of the DUL type RBLs (see points 1-3 in my original post)
> and I think a lot of systems wouldn't meet those preconditions, so I think
> having some way to easily disable all DUL type RBLs is a good idea, and probably
> should be the case by default?
There's no way that this should be disabled by default. It's my belief that
there are more installations that this isn't a problem when things are
configured correctly. Also, there's already way too many complaints about SA
doing a lousy job because the user didn't remove the -L parameter, added by
their friendly neighbordhood package maintainer, from the spamd startup options.
(In reply to comment #4)
> internal_networks was designed to deal with this issue, since it *does* allow
> trust to extend further, without affecting the network boundary used to
> determine which IP to check in the DUL case. The DUL/PBL rules should all be
> using "-lastexternal" accordingly to take that into account.
FWIW, I think extending trusted_networks towards the sender further than
internal_networks is pointless 99.99% of the time (your local DNS cache absorbs
any penalty of a few extra DNS lookups in systems busy enough for the penalty to
be an issue). I recommend that people don't do it. The only place I see
differing trusted/internal configs is on the MSA side where you can't determine
auth from the headers (or POPAuth). With msa_networks now implemented it's not
even necessary in this case anymore.
Also note, and this could have a sizable affect, there's a whole pile of rules
that use X-Spam-Relays-Trusted/Untrusted where they should really use
X-Spam-Relays-Internal/External. Everyone seems to write rules that totally
ignores the possibility of someone extending only trusted_networks towards the
sender.
> > 3. You're willing to penalise SME type customers that run their own mail servers
> > on DSL/dialup lines
>
> We haven't made a definite stand on this, but I think those guys are pretty
> much a lost cause by now. :( SpamAssassin is the _least_ of their worries,
> with pretty much every major ISP (including AOL) blocking them, afaik.
Use DNSBLs, scored for their accuracy, or not (at all). Using the DNSBLs is a
clear winner for me. I think that pretty much anyone who looks at the FN rate
for the non-net scoresets would agree.
In any case... this is really more oriented towards a discussion on one of the
mailing lists rather than in bugzilla.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.